Electronic image sensors such as charge coupled device (CCD) image sensors and active pixel sensor (APS) devices are used in electronic imaging systems for generating an electronic representation of a visual image. APS devices are often fabricated in a Complementary Metal Oxide Semiconductor process and are hence also referred to as CMOS sensors. Typically, these image sensors include a number of light-sensitive pixels (that is, picture elements) arranged in a regular two-dimensional pattern or array of rows and columns, with each individual pixel providing a signal based on the light level of the portion of a scene image projected onto the pixel by a lens.
For considerations of compactness and cost, these image sensors usually include vastly more pixels than analog to digital converters (ADC) to digitize their signals. In order to save space on the sensor chip, it is common practice to provide only enough storage devices to simultaneously read out the pixels of a single row. Consequently, the pixel signals for the complete 2-dimensional sensor cannot be measured or read out simultaneously. Instead, pixel sensor signals are read in a serial fashion. For example, in a CCD having a single ADC, the pixel signals are read out in a raster fashion: pixel-by-pixel within a row, then row-by-row within the array of pixels.
The serial nature of image sensor readout directly controls the rate at which the entire sensor can be read, as limited by the bandwidth of the readout mechanism. For example, if the read-out mechanism of the image sensor can measure 50 million pixels per second, then it would take about one-tenth of a second to read out a 5 megapixel image sensor. Reducing the time required to read the entire image sensor generally requires increasing power consumption for faster read-out, or increasing the size of the image sensor in order to provide additional read-out channels. Neither increased power consumption nor increased size, however, is desirable.
Because it eliminates mechanical components and reduces cost and space requirements, it is a common practice to design an image capture system having no light-blocking shutter. Such systems rely instead on sensor timing that effectively provides an electronic shutter. This timing scheme operates by resetting each photosensor, integrating photo-electrons (or, alternately, holes), and then reading out the photosensor signal in an ordered sequence. The reset step can be accomplished by transferring residual charge from a photodiode to associated floating diffusion circuitry and then discarding the residual charge. During exposure, the photo-electrons accumulate in the photodiode for the prescribed integration time, at which point the charge signal is transferred into the floating diffusion. In CMOS devices, the charge signal is converted to a voltage. The associated voltage is then stored in a memory device such as a capacitor.
If the sensor has sufficiently low dark current and sufficiently good light shielding for the floating diffusion, then the transferred charge need not be read out immediately. Under these conditions, it would be possible to transfer the charge from all pixels at once into their respective floating diffusions and then to wait for a short time as the sequential read-out sequence progressively processes the signals, row by row. Of course, for such a global transfer to work, each pixel would also need to have its own light-shielded floating diffusion.
An alternative image sensor readout arrangement, provided particularly by APS image sensors, allows exposure and readout of the image sensor to occur progressively row-by-row across the rows of the image sensor. This “rolling shutter” sequence avoids the differential exposure problem exhibited in the interlaced fields of a CCD, making the exposure for each row extend for the same length of time. As an additional advantage, the rolling shutter sequence simplifies sensor component design, since shielded storage is not required for each pixel. However, since the exposure for each row is independent from the exposures of the other rows and occurs in a sequential (or rolling) fashion with the exposures of the other rows, each successive row captures its portion of a scene image at a slightly later time than the preceding row. Consequently, relative motion between the scene (or elements of the scene) and the image sensor causes objects within the scene to appear distorted in the image captured by the image sensor. This effect, termed image “shear”, is characteristic of rolling shutter arrangements. For example, if such a so-called rolling shutter or electronic focal plane shutter image sensor is used to capture an image of an automobile moving horizontally, the automobile moves relative to the image sensor as each row of the captured image is exposed and read out, so that each row of the captured image shows the vehicle at a different position. This can cause round car tires to appear to be somewhat oval, and distort rectangular car windows to appear as parallelograms. This distortion due to motion is a direct consequence of the amount of time required to read out all the rows of the image sensor. If the rows can be read at a faster rate, then this distortion can be reduced. As noted previously, however, increasing the readout rate generally requires an undesirable increase in cost and power consumption for the image sensor.
For silicon-based image sensors, the pixel components themselves are broadly sensitive to visible light, permitting unfiltered pixels to be suitable for capturing a monochrome image. For capturing color images, a two-dimensional pattern of filters is typically fabricated on the pattern of pixels, with different filter materials used to make individual pixels sensitive to only a portion of the visible light spectrum. An example of such a pattern of filters is the well-known Bayer color filter array (CFA) pattern, as described in U.S. Pat. No. 3,971,065. Though the Bayer CFA has advantages for obtaining full color images under typical conditions, however, this solution has been found to have its drawbacks. Although filters are needed to provide narrow-band spectral response, any filtering of the incident light tends to reduce the amount of light that reaches each pixel, thereby reducing the effective light sensitivity of each pixel and reducing pixel response speed.
As solutions for improving image capture under varying light conditions and for improving overall sensitivity of the imaging sensor, modifications to the familiar Bayer pattern have been disclosed. For example, commonly assigned U.S. Patent Application Publication No. 2007/0046807 entitled “Capturing Images Under Varying Lighting Conditions” by Hamilton et al. and U.S. Patent Application Publication No. 2007/0024931 entitled “Image Sensor with Improved Light Sensitivity” by Compton et al. both describe alternative sensor arrangements that combine color filters with panchromatic filter elements, spatially interleaved in some manner. With this type of solution, some portion of the image sensor detects color; the other panchromatic portion is optimized to detect light spanning the visible band for improved dynamic range and sensitivity. These solutions thus provide a pattern of pixels, some pixels with color filters (providing a narrow-band spectral response) and some without (unfiltered “panchromatic” pixels or pixels filtered to provide a broad-band spectral response).
Using a combination of both narrow- and wide-spectral band pixel responses, image sensors can be used at lower light levels or provide shorter exposure times. See, for example, Sato et al. in U.S. Pat. No. 4,390,895, Yamagami et al. in U.S. Pat. No. 5,323,233, and Gindele et al. in U.S. Pat. No. 6,476,865.
Configurations with both color and panchromatic pixels can help to compensate for some of the problems caused by color filtering, particularly with respect to reduced light sensitivity and motion blur. However, when using conventional pixel readout timing sequences, correction of motion problems, caused by the time delay between exposure intervals for panchromatic and color capture over the same portion of the image sensor, requires that motion estimation or other compensation techniques be implemented. Thus, sophisticated software can be required for taking full advantage of implementations using both color and panchromatic pixels, adding cost and complexity to imaging device designs.
There is, then, a need for improved readout methods that reduce or eliminate the requirements for motion estimation and compensation when using a digital imaging sensor array.